基于变压器的模型在多个领域和任务上显示了它们的有效性。自我注意力允许将所有序列元素的信息结合到上下文感知表示形式中。但是，全球和本地信息必须主要存储在相同的元素表示中。此外，输入序列的长度受到自我注意的二次计算复杂性的限制。在这项工作中，我们提出并研究了一个记忆启动的片段级循环变压器（复发记忆变压器）。内存允许借助复发的帮助存储和处理本地和全局信息，并可以在长序列的段之间传递信息。我们通过将特殊的内存令牌添加到输入或输出序列中，实现了一个内存机制，无需更改变压器模型。然后，对变压器进行了训练，以控制内存操作和序列表示处理。实验的结果表明，我们的模型与Transformer-XL在语言建模上的较小内存大小上的表现相同，并在需要更长序列处理的任务方面胜过它。我们证明，将内存令牌添加到TR-XL可以提高IT性能。这使得反复的内存变压器成为需要学习长期依赖性和内存处理中的通用性（例如算法任务和推理）的应用程序的有前途的体系结构。

In atomistic simulations of solids, ability to classify crystal phases and lattice defects in the presence of thermal fluctuations is essential for gaining deeper insights into the simulated dynamics. The need for accurate and efficient characterization methods is especially acute in presently emerging large-scale simulations of multi-phase systems far from equilibrium. Taking the perspective that delineating order and disorder features from ubiquitous thermal vibrations is akin to extracting signal from noise, we consider classification of ordered phases and identification of disordered crystal defects to be fundamentally the same problem and address them both with a unified approach: a denoising score function that removes thermal noise and recovers any underlying crystalline order-disorder. Built on a rotationally equivariant graph neural network (NequIP), the denoiser was trained entirely with synthetically noised structures and requires no simulation data during training. To demonstrate its denoising capabilities, the denoiser is shown to effectively remove thermal vibrations of BCC, FCC, and HCP crystal structures without impacting the underlying disordered defects, including point defects, dislocations, grain boundaries, and liquid disorder. In particular the denoiser was applied to two relatively complex MD simulations that present practical challenges: a Cu solidification trajectory involving a polymorphic nucleus, and a trajectory of BCC Ta undergoing plastic deformation resulting in dislocation networks and point defect clusters. In both cases the denoiser facilitates or trivializes the subsequent characterization of the order-disorder features. Lastly, we outline future work to extend our denoising model to more complex crystal structures and to multi-element systems.